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ABSTRACT 


Three  pairs  of  parallel  R&D  projects  are 
examined.   The  data  analyzed  were  gathered  by  means 
of  Solution  Development  Records--a  form  which  pro- 
vides a  weekly  estimate  of  the  probability  of  adop- 
tion of  the  approaches  under  consideration  as  pos- 
sible solutions  to  a  technical  problem.   It  is  found 
that  the  longer  an  approach  is  indicated  by  these 
forms  to  be  in  a  favored  position,  the  more  diffi- 
cult it  is  to  reject.  Furthermore,  the  number  of  al- 
ternative technical  approaches  considered  bears  a 
relation  to  judged  solution  quality.   Groups  pro- 
ducing higher-rated  solutions  generated  fewer  ap- 
proaches during  the  course  of  the  project,  and 
they  more  closely  approach  an  ideal  strategy  of 
approaches  off  on  a  two-at-a-t ime  basis  than  do  their 
poorer  performing  rivals. 


The  research  and  development  process  provides  a  challenge  to  the 
behavioral  scientist  who  wants  to  study  the  process  of  human  problem  sol- 
ving in  vivo.   Engineers  and  scientists  i n  R  &  D  laboratories  devote  their 
careers  to  the  solving  of  rather  sophisticated  problems,  and  the  study  of 
their  behavior  should  tell  us  much  about  the  higher  mental  processes. 

Such  a  study,  however,  presents  the  researcher  with  two  difficulties 
which  he  must  resolve.   The  first  of  these  is  access.  Access  must  be 
gained,  not  only  to  the  laboratory  under  study,  but  to  the  subtle  men- 
tal processes  of  individual  scientists.   This  is  a  standard  problem  in 
psychology,  but  the  real-life  environment  serves  to  compound  it. 
Observable  indices  that  in  some  way  provide  valid,  reliable  estimates 
of  the  actual  cognitive  data  must  be  established.   In  addition  to 
validity  and  reliability  the  indices  must  further  be  highly  efficient 
in  terms  of  the  time  required  on  the  part  of  subjects.   The  latter  re- 
quirement is  imposed  by  the  field  situation,  and  is  absolutely  essen- 
tial to  the  recruitment  of  cooperative  subjects. 

The  second  difficulty  to  be  faced  is  that  of  problem  comparability 
and  solution  evaluation.   In  the  psychological  laboratory,  this  is  easily 
managed  by  assigning  the  same  problem  to  large  numbers  of  college  so- 
phomores.  The  substance  of  the  problem  is  controlled  so  that  a  valid 
comparison  can  be  made  of  the  subjects'  approaches.   In  addition,  the 
problems  are  generally  of  a  type  which  has  but  one  correct  solution.   The 
sample  of  problems  can  then  be  split  into  those  having  correct  and  incor- 
rect solutions  and  approaches  may  be  related  to  success  rate. 
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METHOD 

While  we,  as  researchers,  have  not  reached  the  level  of  affluence 
where  we  can  h i re  a  number  of  engineers  and  assign  them  the  same  pro- 
blem, there  are  organizations  such  as  N.A.S.A.  and  the  Department  of 
Defense  who  can.  For  this  reason,  instances  in  which  two  or  more 
R&D  laboratories  have  been  awarded  a  contract  to  perform  the  same 
preliminary  design  or  research  investigation  were  sought  out  for  study, 

At  the  present  time,  about  a  dozen  sets  of  two  or  three  parallel 
projects  have  been  located  and  studied.   This  paper  will  present  the 
results  of  an  analysis  of  three  of  the  earliest  completed  sets. 

Once  a  parallel  project  has  been  located,  its  work  statement  is 
obtained  and  analyzed  and  factored  into  a  reasonable  number  of  sub- 
problem  areas  (generally  subsystems).   The  breakdown  is  then  checked 
with  the  technical  person  who  prepared  the  work  statement,  and  data 
collection  forms  based  upon  it  are  designed.   After  all  data  have  been 
collected  from  the  contractors,  the  technical  monitor  is  revisited 
and  asked  to  provide  a  confidential  evaluation  of  each  lab's  perfor- 
mance on  each  subproblem.   Data  are  gathered  by  four  means:   (I) 
time  allocation  forms,  indicating  the  amount  of  time  each  engineer 
spends  on  the  job  in  several  activity  categories;  (2)  before  and  after 
interviews  with  individual  engineers:  (3)  periodic  tape  recorded  pro- 
gress reports  by  the  project  manager;  and  (4)  solution  development  re- 
cords. 

The  Solution  Development  Record,  which  is  the  principal  source  of 
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the  data  presented  in  this  paper,  is  a  research  tool  which  provides  a 
record  over  time  of  the  progress  of  an  individual  engineer  or  group  of 
engineers  (or  scientists)  toward  the  solution  of  a  technical  problem. 
The  lead  engineer  responsible  for  each  subproblem  is  asked  to  provide 
a  weekly  estimate,  for  each  alternative  approach  under  consideration,  of 
the  probabi  I  ity  that  it  wi  I  I  be  final  ly  chosen  as  the  solution  to  that 
subprob I  em. 

Referring  to  Figure  I,  i f  at  some  point  in  the  design  the  respondent 
were  considering  two  technical  approaches  to  rendezvous  at  Uranus,  and 
he  were  completely  uncommitted  between  the  two,  he  would  circle  0.5 
for  each,  as  shown.   Eventually  as  the  solution  progresses,  one  alter- 
native wi  I  I  attain  a  I .0  probabi  I  ity  and  the  others  wi  I  I  become  zero. 
By  plotting  the  probabilities  over  time,  we  obtain  a  graphic  record  of 
the  solution  history.   Alternative  approaches  are  identified  from  the 
contract  work  statement,  when  so  specified,  or  from  the  responsible  en- 
gineer when  he  is  interviewed  prior  to  beginning  the  task.   Blank 
spaces  are  always  provided  so  that  new  approaches  may  be  reported  as  they 
arise.   In  cases  where  a  respondent  believes  there  is  some  probability 
of  choosing  an  approach  which  he  cannot  clearly  specify  at  the  time,  he 
is  instructed  to  assign  a  probability  to  an  approach  which  he  may  call 
"other." 

A  copy  of  the  form  is  mailed  every  week  to  each  respondent.  The 
forms  are  sufficiently  flexible,  so  that  new  alternatives  may  be  incorpora- 
ted, old  ones  dropped,  and  in  instances  in  which  an  early  solution  is 
reached  and  "frozen",  subproblems  at  the  next  level  may  be  substituted. 


The  Sol'ution  Development  Record,  by  economizing  on  the  respondent's 
time,  provides  a  quite  efficient  record  of  a  problem  history.   When  the 
project  is  completed,  each  respondent  is  presented  with  a  time-plot  of 
his  probability  estimates,  and  is  interviewed  at  some  length  to  detei — 
mine  causes  and  effects  of  design  changes  reflected  in  this  record. 
The  plot  thus  provides  a  stimulus  to  the  man's  memory  and  assists  the 
investigator  in  gathering  a  detailed  record  of  each  project. 

The  three  projects  under  consideration,  involved  in  the  following 
general  problems: 

1.  The  design  of  the  reflector  portion  of  a  rather  large 
and  highly  complex  antenna  system  for  tracking  and 
communication  with  space  vehicles  at  very  great  dis- 
tances. 

2.  The  design  of  a  vehicle  and  associated  instrumentation 
to  roam  the  lunar  surface  and  gather  descriptive 
scient i  f  i  c  data. 

3.  An  investigation  of  passive  methods  for  transfer  of  modu- 
lation between  two  coherent  light  beams. 

Only  the  first  of  these  will  be  discussed  in  any  detail.   All  three 
are  considered  in  the  aggregate  statistics  of  our  data. 

RESULTS 

The  plot  of  Solution  Development  Record  points  over  time  (Figure  2) 


FIGURE  I 

MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 
Solution  Development  Record 

Manned  Uranus  Landing  in  an  Early  Time  Period  Study 
General  United  Aerospace  Corporation 
Name  Date ____ 


Estimate  of  Probabl lity  that  Alternative  will 
be  Employed 

Subproblem  #1:  Method  of 
rendezvous  at  Uranus 

Alternative  approaches: 

orbital  rendezvous  mi ss ion     0  0.1  0.2  0.3  0.4  (p.s)  0.6  0.7  0.8  0.9  1.0 
with  excursion  vehicle 

orbital  rendezvous  mission     0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 
without  excursion  vehicle 


direct  mission  0  0.1  0.2  0.3  0.4  {0J)   0.6  0.7  0.8  0.9  1.0 
0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 


Subproblem  #2:   Design  of  the 
electrical  power  supply  subsystem 
for  the  space  vehicle 

Alternative  approaches: 

hydrogen-oxygen  0  0.1  0.2  0.3  0.4  0.5  (Q)   0.7  0.8  0.9  1.0 

fuel  eel  I 


K0H  fuel  cell  0  0.1  0.2  (CK3)  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Rank ine  cycle  fast  reactor  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Rank ine  cycle  thermal  reactor  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Brayton  cycle  reactor  0  (oj)  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

___  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

.  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 


provides  a  rather  interesting  perspective  on  the  history  of  a  project 
and  illustrates  the  intimate  relation  between  technical  information  in- 
puts and  problem  solving  process. 

We  can  see  here  the  approaches  followed  by  two  engineering  groups 
(labelled  Lab  A  and  Lab  B)  in  the  design  of  the  reflector  surface  for 
an  antenna.   While  both  teams  ultimately  decided  upon  the  same  general 
approach,  they  arrived  there  by  quite  different  routes.   A  brief  sum- 
mary of  the  history  of  the  two  solution  processes  for  this  subproblem 
will  explain  much  of  this  difference,  and  will  provide  some  background 
substance  for  much  of  the  data  to  be  presented  later. 

Rather  early  in  the  study,  as  indicated  by  flag  notes  A- I  and  B-2, 
the  customer  agency  supplied  both  contractors  with  the  results  of  an 
experiment  to  determine  the  wind  loadings  which  would  be  experienced  by 
the  antenna.   As  a  result,  approach  p  rose  in  favor  at  both  labs.   Prior 
to  this  time,  Lab  A  showed  considerable  vacillation  among  the  three  al- 
ternatives; Lab  B,  during  this  early  period,  conducted  an  intensive  lit- 
erature search  but  failed  to  uncover  any  evidence  of  empirical  or  ana- 
lytical work  having  been  done  with  flat  aerodynamic  shapes  at  the  low 
air  speeds  under  consideration. 

At  A-2  Lab  A's  aerodynamic  staff  reported  wind  load  moments  for  ap- 
proach "V  to  be  about  twice  that  for  o(  or  fd>     .      At  the  same  time,  the 
electromagnetic  staff  reported  that  approaches  (j\    and  p    satisfactorily 
met  electrical  requirements  up  to  3  gigacycles. 

An  electromagnetic  analysis  by  Lab  B  shows  that  approach  c{    failed  to 
satisfy  the  electrical  performance  specification.   This  is  indicated  by 
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DESIGN  OF  ANTENNA  RADIATION  SUBSYSTEM 


flagnote  B-3.  During  the  same  period  (B-4),  this  laboratory  at- 
tempted to  extrapolate  from  previously  acquired  data  (an  earlier 
antenna)  to  estimate  wind  loads. 

About  the  twelfth  week  (A-3)  Lab  A  conducted  a  wind  tunnel  study, 
which  showed  that  approach  p  resulted  in  a  wind  torque  considerably 
larger  than  that  predicted  by  the  customer's  data  (A-l).   Since  ap- 
proach <f\   did  not  perform  as  well  as  ft ,  electrically,  one  of  three 
things  now  had  to  be  done.   E'ther  the  aerodynamic  specification 
had  to  be  relaxed,  a  penalty  in  electrical  performance  accommodated, 
or  a  new  alternative  meeting  both  the  aerodynamic  specifications  and 
the  electrical  performance  level  of  approach  d   had  to  be  generated. 
The  latter  possibility  did  not  present  itself,  so  negotiations  with 
the  customer  over  specifications  were  pursued. 

The  customer  subsequently  allowed  a  relaxation  of  the  electrical 
specification  (A-4)  and  approach  A   rose  in  favor.   The  change  in  speci- 
fication was  provided  to  Lab  B  as  well,  but  there  was  no  consequent 
change  in  the  probability  level  of  approach  A  .      This  undoubtedly 
resulted  from  the  fact  that  B  did  not  have  as  complete  information 
as  A  regarding  approach  p.      The  brief  drop  in  p's  position  at 
Lab  B  (B-5)  was  a  result  of  some  doubts  which  this  lab  had  concerning 
the  feasibility  of  the  approach,  but  as  far  as  can  be  determined  this 
was  not  based  upon  hard  data.   Following  this  brief  period  of  skepti- 
cism, p  rose  rapidly  to  a  1.0  level  (B-6)  and  was  further  established 
there  when,  as  indicated  by  B-7,  information  concerning  special  fabri- 
cating machinery  became  available. 
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At  Lab  A,  meanwhile,  approach  Rencountered  some  difficulty  with 
the  cost  analysts.   And  we  see  a  resulting  dropoff  at  note  A-5.   Lab 
A  remained  indifferent  between  ,j(  and  p  for  quite  some  time  while  trade- 
off studies  were  pursued  (A-6).   Numerous  contacts  were  made  with  ven- 
dors to  determine  the  costs  associated  with  various  elaborations  of 
the  two  approaches. 

Finally,  at  A-7,  information  was  obtained  from  the  Weather  Bureau 
which  allowed  a  20$  reduction  in  the  wind  loading  specification.  This 
information  was  instrumental  in  Lab  A's  decision  to  adopt  approach  p  . 

The  work  statement  for  the  subsystem  under  consideration  suggested 
three  technical  approaches,  and  these  were  the  only  ones  considered 
by  the  two  teams.   That  this  is  not  always  the  case  can  be  demonstrated 
by  a  look  at  another  of  the  antenna  subsystems  (Figure  3). 

Here  we  see  design  histories  for  the  position  feedback  subsystem. 
Space  does  not  permit  a  detailed  discussion  of  the  protocol  for  this 
subproblem,  so  we  will  merely  point  out  the  introduction  of  new  ap- 
proaches during  the  course  of  the  project.   The  work  statement  for 
both  labs  suggested  approaches  c{    ,p    and  J    ■      Both  rejected  these, 
however,  and  generated  two  new  approaches  each  ( <g  ,  <f  ,  3  and  y  )  .   In 
both  labs  one  of  the  new  approaches  resulted  from  difficulties  incur- 
red by  the  currently  preferred  approach;  the  other  resulted  from  re- 
ceipt of  new  information,  and  was  independent  of  the  state  of  approaches 
currently  under  consideration. 
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DESIGN  OF  SUBSYSTEM   TO   DETERMINE 
ANTENNA      POSITION 


Decay  Time  for  Rejected  Approaches 

There  are   a  number  of  ways  in  which  the  data  contained  in  these 
plots  can  be  analyzed.   Looking  at  the  time  required  for  rejection  or 
introduction  of  approaches,  we  see  that  old  ideas  are  not  dispensed 
with  very  abruptly,  nor  do  new  ideas  rise  very  rapidly.   Once  an  R  &  D 
group  has  committed  very  much  time  to  the  design  of  a  particular  ap- 
proach, it  is  not  easily  convinced  that  this  approach  should  be  dropped. 
Analysis  of  31  instances  shows  that  a  once  dominant  idea  takes  about 
six  or  seven  weeks  on  the  average  to  be  completely  rejected.   Domin- 
ance is  defined  as  being  in  the  preferred  position  in  the  probability 
plot.   Decay  time  is  measured  in  two  ways: 

1.  if  the  old  approach  is  replaced  by  a  new  idea,  decay  time 
is  the  period  between  introduction  of  the  new  idea  and  the 
point  at  which  the  old  approach  reaches  a  probability  of 
zero  and  stays  there. 

2.  if  the  old  approach  is  replaced  by  one  of  the  approaches 
initially  considered,  decay  time  is  the  period  between  the 
point  at  which  the  two  approaches  trade  dominance  or  reach  a 
tied  position,  and  the  point  at  which  the  rejected  idea  reaches 
zero  and  stays  there. 

There  is  a  clear  relation  between  decay  time  and  time  in  dominance. 
(r=0.54,  p^O.OOl)   The  longer  an  idea  is  preferred,  the  more  inertia  is 


Pearson  Product  Moment  Correlation  Coefficient  (r)  with  a  probabi- 
lity  of  occurrence  (p)  of  a  coefficient  this  high,  under  the  null  hypothe- 
sis of  no  relation,  less  than  0.001.   The  correlation  coefficient  varies 
in  magnitude  from  zero  to  one  and  is  an  index  of  goodness  of  fit  to  a  least- 
squares  straight  line. 


built  up,  and  the  less  easily  is  it  rejected.   This  is  independent  of 
the  absolute  level  of  preference  (r=0.03)  and  of  the  number  of  other 
alternatives  being  considered  (r=-0.04). 

As  engineers  invest  time  and  effort  in  the  formulation  and  deve- 
lopment of  a  technical  approach,  they  become  more  and  more  committed 
to  that  approach,  and  hence  more  resistant  to  d i sconf i rmi ng  informa- 
tion.  An  analogy  may  be  drawn  here  to  Bruner's  (1957)  paradigm  of 
the  decision  sequence  in  perception.   During  the  initial  consideration 
of  a  new  idea,  the  engineer  is  relatively  "open"  to  stimulation  from 
his  information  environment.   As  the  amount  of  confirming  information 
increases,  he  raises  his  probability  estimate  for  the  alternative  and 
nodifies  his  search  behavior.   The  "openness"  to  new  information  de- 
creases sharply  now.   The  longer  he  retains  the  approach  in  a  dominant 
position  with  respect  to  its  alternatives,  the  more  he  attenuates 
his  search  behavior.   At  this  point  he  enters  a  stage  quite  analogous  to 
Bruner's  final  stage  in  perceptual  identification,  where: 

"...openness  to  additional  cues  is  drastically  reduced, 
and  either  normalized  or  'gated  out.'   Experiments... 
suggest  that  once  an  object  has  been  categorized  in  a 
high-probability,  good-fit  category,  the  threshold  for 
recognizing  cues  contrary  to  this  categorization  increases 
by  almost  an  order  of  magnitude."   (p.  131) 

What  is  being  suggested  here  is  that  an  engineer  tends  to  develop 

a  similar  threshold  as  be  becomes  committed  to  a  technical  approach,  and 

that  this  threshold  severely  inhibits  the  effect  of  information  which 

should  tell  him  that  the  approach  is  defective  in  some  way.   In  addition, 

it  appears  to  gate  out  information  related  to  new  alternative  approaches. 


Examination  of  protocol  data  reveals  that  the  vast  majority  of  new 
alternatives  introduced  during  the  course  of  a  problem  result  from  an 
active  search  resulting  from  some  defect  in  the  currently  preferred  ap- 
proach.  Seldom  does  a  new  idea  enter  the  system  solely  on  its  own 
merit . 

From  studies  of  perception,  and  the  receipt  by  an  engineer  of  new 
technical  information  is  certainly  a  complex  form  of  perception*  there 
is  a  great  deal  of  support  for  this  threshold  hypothesis.   McGinnies 
(1949),  has  demonstrated  a  physiological  defense  mechanism  in  the  human 
organism,  and  a  tendency  toward  higher  recognition  thresholds  for  ta- 
boo as  compared  to  neutral  words.   Bruner  and  Postman  (1949)  have  de- 
monstrated a  tendency  to  fixate  upon  an  early  perceptual  hypothesis 
and  to  subsequently  resist  the  recognition  of  d i sconf i rmi ng  information. 
Postman,  Bruner  and  Walk  (1951)  have  shown  the  operation  of  such  a 
threshold  to  gate  out  those  signals  which  would  tend  to  disconfirm  a 
perceptual  hypothesis. 

Relatively  few  data  are  available  in  the  purely  cognitive  area. 
One  may,  however,  hypothesize  that  the  operation  of  threshold  develop- 
ment in  this  context  is  a  result  of  the  progressive  buildup  of  cogni- 
tive organization,  i.e.,  of  the  multiplexity  and  i nterconnectedness  of 
the  engineer's  cognition  of  his  design. 

This  can  be  illustrated  by  considering  an  engineer  designing,  for 
example,  a  power  supply.   At  the  start  of  the  problem  he  may  know  little 
concerning  specific  alternatives,  except  that  they  are  different  and 
have  certain  very  general  characteristics  which  may  or  may  not  be  ex- 
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pected  to  fulfill  the  needs  of  the  problem.   As  he  gains  more  informa- 
tion concerning  an  alternative,  ho  shifts,  gradually,  from  thinking  of 
it  as  simply  a  'radioisotope  power  supply"  and  comes  to  think  of  it 
as  a  "radioisotope  thermionic  supply":   employing  Cm  242;  having  a  cer- 
tain amount  and  kind  of  lead  shielding;  weighing  four  pounds;  genera- 
ting twelve  watts  of  effective  power;  and  having  an  operating  temperature 
of  1500  K.   In  addition  to  this  increase  in  mu I t i d imens iona I i ty  ormulti- 
plexity  of  the  concept,  it  becomes  progressively  more  interconnected  or 
integrated  into  the  design  of  the  system.   Several  other  subsystem  de- 
signs will  usually  be  dependent  upon  the  solution  to  a  particular  sub- 
problem  such  that  there  is  a  great  deal  of  interdependence  among  the 
subsystems.   As  the  design  progresses  these  i nterdependencies  become 
stronger  and  more  pronounced.   Various  decisions  in,  for  example,  the 
instrumentation  subsystem  of  a  space  vehicle  are  dependent  upon  the 
power  supply  design.   Changes  in  that  design  imply  changes  in  instru- 
mentation which  in  turn  imply  changes  elsewhere  and  so  on.   Such 
changes  become  more  difficult  to  make  as  time  progresses. 

All  of  this  is  reflected  to  some  extent  in  the  cognitive  struc- 
ture of  the  engineer  designing  the  power  supply.  As  the  organization 
of  his  cognitive  map  progressively  increases,  it  becomes  more  and  more 
difficult  for  him  to  envisage  fundamental  changes.   It  seems  perfectly 
reasonable  then  that  his  cognitive  system  should  become  more  resistant 
to  change  and  that  a  threshold  against  information  implying  fundamental 
change  should  build  up  as  part  of  this  process.   As  a  solution  alterna- 
tive becomes  more  embedded  in  a  growing  cognitive  organization,  it  deve- 
lops a  greater  resistance  to  change. 
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This  phenomenon  can  be  seen  quite  often  in  the  protocol  data.   An 
engineer  decides  rather  early  to  pursue  a  particular  alternative  and 
formulates  his  design  to  a  considerable  degree  of  sophistication.   Then 
he  is  confronted  with  information  that  something  is  wrong  with  one  of 
the  dimensions:   perhaps  it  begins  to  appear  to  be  too  costly,  or  the 
design  has  become  too  complex  and  has  sacrificed  reliability.   This 
information  may  be  ignored  at  first,  or  an  inordinate  amount  of  effort 
may  be  expended  in  attacking  this  aspect  of  the  problem  (which  might 
readily  appear  intractible  to  an  equally  experienced  outsider).   At 
length  he  comes  around  and  accepts  the  necessity  for  basic  modification 
in  the  design,  but  the  process  usually  requires  a  considerable  amount 
of  time  and  energy  for  its  accomplishment. 

In  all  of  this,  it  is  not  our  intent  to  deny  the  affective  com- 
ponent.  Despite  the  norm  of  dispassionateness,  engineers  do  become 

committed  to  their  own  designs.   This  commitment  undoubtedly  develops 

2 
with  time.    But  such  commitment  should  also  be  a  function  of  the 

uniqueness  of  the  design,  and  in  the  instances  under  examination  few 

of  the  rejected  approaches  could  be  considered  to  be  really  very  novel. 

Furthermore,  the  engineers  were  in  a  position  to  claim  full  credit  for 

the  development  of  the  replacement  idea  and  had  little  to  lose  and 

actually  something  to  gain  through  rejection  of  the  old  ideas. 


2 
For  a  fascinating  discussion  of  this  problem,  the  reader  is  re- 
ferred to  an  article  by  T.  C.  Chamber  I  i n  originally  published  in  1890 
and  reprinted  in  Sci  ence,  1 48,  7  May  1965,  pp.  754-759. 


Number  of  Alternative  Approaches  Considered 

Engineers  considered  from  one  to  eight  approaches  to  a  subproblem 
with  a  median  of  three.  (Teble  I).   Two  of  the  three  approaches  were 
considered  from  the  beginning  of  the  project,  with  a  median  of  one 
new  approach  arising  during  the  project's  course. 

TABLE  I 

Median  Number  of  Alternative  Approaches  Considered 
(20  Subproblem  Pairs;  Three  Parallel  Projects) 


median  range 


total  3  1-8 

initial  2  1-5 

add  it iona I  I  0-5 

The  ideas  which  come  up  later  in  the  project  have  a  success  rate  of 
about  15$,  compared  with  a  success  rate  of  39.5$  for  initially  considered 
approaches.   Eighty  per  cent  of  the  approaches  finally  adopted  were  among 
those  initially  considered.   Only  20  per  cent  of  solutions  originated 
during  the  course  of  the  study. 

Considering  once  again  the  total  number  of  alternative  approaches 
considered  by  the  engineers,  it  was  hypothesized  that  this  number  should 


bear  a  direct  relation  to  solution  quality.   More  specifically,  the 
hypothesis  stated  that  the  greater  the  number  of  alternative  approaches 
considered  to  a  subprob lem,  the  better  the  quality  of  the  solution  to 
that  subproblem. 

On  eight  of  the  20  subproblem  pairs  in  the  sample,  relative  eval- 
uations of  the  solutions  were  obtained  from  responsible  technical  moni- 
tors in  the  government  agency.   In  the  remaining  twelve,  scores  were 
either  tied  or  no  evaluation  was  available.   This  relative  evaluation 
permits  the  use  of  simple  tests  between  the  number  of  alternatives  con- 
sidered and  judged  quality  of  the  solution. 

Not  only  is  the  hypothesis  concerning  total  number  of  alternatives, 
not  supported  by  the  data,  but  as  can  be  seen  from  Table  II,  the  rela- 
tion is  in  the  opposite  direction. 

Engineers  submitting  highei — rated  solutions  actually  consider 
fewer  alternatives  than  their  lower-rated  competitors.   This  not  only 
contradicts  the  hypothesis  but  appears  to  be  in  direct  conflict  with 
previous  findings.   Allen  and  Marquis  (1964)  in  a  study  of  two  R&D 
proposal  competitions  found  that  teams  considering  a  larger  number  of 
approaches  to  a  proposal  subproblem  were  better  able  to  overcome  ne- 
gative transfer  from  related  prior  experience.   Furthermore,  going  back 
to  the  data  from  the  earlier  study,  it  is  found  that  proposal  teams  sub- 
mitting successful  solutions  had  considered  2.13  approaches,  on  the  aver- 
age, compared  with  a  mean  of  1.71  for  teams  with  unsuccessful  solutions. 
"Success"  was  determined  by  technical  eva I uators  in  the  customer  agency. 
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Since  proposal  competitions  represent  a  relatively  early  phase  in 
the  R&D  process,  the  possibility  remains  that  the  relation  between 
number  of  alternatives  and  quality  is  a  function  of  problem  phase,  and 
that  this  relation  shifts  direction  as  the  project  progresses  beyond 
its  initial  phase.   To  test  this  possibility  the  number  of  approaches 
considered  initially  are  separated  out  and  compared.  As  can  be  seen 
in  Table  11^   this  difference  is  still  in  a  direction  which  favors  the 
lower-rated  subproblems,  but  is  reduced  in  magnitude  and  far  from  statis- 
tical significance.   The  last  line  in  the  table  shows  that  the  real  dis- 
tinction between  the  two  sets  lies  in  the  number  of  alternatives  generated 
when  a  team  encounters  trouble  with  one  of  their  initial  approaches,  and 
this  difficulty  is  reflected  in  a  greater  number  of  alternatives  generated 
during  the  later  stages  of  the  problem  and  in  the  quality  of  the  solu- 
tion. 

TABLE  I  I 

Mean  Number  of  Alternative  Approaches 
Considered  per  Subproblem 
(Eight  Subproblem  Pairs) 


subproblems  with     subproblems  with  p 

higher-rated         lower-rated  (as  determined 

solutions           solutions  by  t  test) 

4.37  0.06 

2.63  0.10 

1.75  0.05 


total 

2.88 

initial 

2.38 

add  i  t  i  ona 1 

0.50 
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Number  of  Alternative  Approaches  Considered  Simultaneously 

In  Figures  2  and  3  it  may  be  noted  that  there  is  a  distinct  tendency 
for  engineers  to  consider  but  two  or  three  alternatives  at  any  given  time. 
During  the  course  of  a  problem,  they  may  consider  as  many  as  eight  altei — 
natives,  but  they  seldom  report  considering  all  of  these  at  once.   The 
mean  number  of  alternatives  simultaneously  considered  ranges  from  one 
to  five  with  a  mean  of  2.20.   These  statistics  are  based  upon  those  weeks 
prior  to  the  point  at  which  one  of  the  approaches  attains  a  probability 
of  1.0  and  stays  there.   Inclusion  of  reports  received  beyond  that  point 
would  obviously  lower  the  mean  value. 

Figure  4  provides  a  relative  frequency  distribution  of  simultaneously 
considered  alternatives  and  is  based  upon  weekly  reports.   The  strong 
central  tendency  around  two  alternatives,  and  the  sharp  cutoff  above 
three  can  be  seen  quite  readily  from  this  figure. 

Since  evaluations  of  solution  quality  are  available  for  eight  of 
the  20  subproblem  pairs,  a  comparison  was  made  between  the  means  for 
groups  submitting  lower  and  higher-rated  solutions.   On  the  average 
those  producing  higher  rated  solutions  considered  2.08  alternatives  at  a 
time;  lower  rated  teams  considered  2.42  alternatives  at  a  time.   The  dif- 
ference between  means  is  statistically  significant  at  the  0.001  level. 

Frequency  distributions  of  alternatives  (figure  5)  for  the  rela- 
tively higher  and  lower  rated  performances  show  that  those  teams  turning 
in  better  performances  come  much  closer  to  the  ideal  of  two  at  a  time. 
In  fact,  8\%   of  the  time  they  report  considering  only  two  alternatives 
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(58J?  for  lower-rated)  and  they  never  report  considering  more  than  three 
alternatives.  A  Kolmogorov-Smi rnov  two-sample  test  (Siege  I  1956)  re- 
jects the  null  hypothesis  that  these  two  samples  are  drawn  from  the 
same  universe  (p<O.OOI). 

It  thus  appears  that  more  successful  engineers  differ  from  their 
brethren  who  are  poorer  problem-solvers  in  the  manner  in  which  they 
trade  off  and  consider  alternative  solution  possibilities.  Further, 
the  principal  difference  appears  to  be  the  tendency  toward  limiting 
their  consideration  to  two  alternatives  at  a  time. 

Why  should  this  be  so?  A  number  of  investigators,  (Miller,  1965; 
Hayes,  1962;  Santa  Barbara  and  Pare,  1965)  have  demonstrated  a  finite 
limitation  on  human  information  processing  capacity.   Neimack  and 
Wagner  have  shown  further  that  amount  of  information  gathering  is  a 
linear  function  of  log  2 r>  (where  n  is  the  number  of  possible  solutions). 
While  their  measure  of  information  gathering  is  the  number  of  discrete 
steps  taken  in  a  rather  limited  type  of  problem,  it  does  seem  clear 
that  the  amount  of  information  which  must  be  processed  should  increase 
with  the  number  of  alternatives  being  considered.   Hayes  (1962)  has 
shown  quality  of  solution  in  a  decision-making  context  to  be  indepen- 
dent of  the  amount  of  information  (number  of  characteristics)  provided 
on  each  alternative,  but  to  be  significantly  decreased  as  the  number 
of  alternatives  is  increased  from  four  to  eight.   Increasing  the  number 
of  characteristics  increases  the  dimensionality  of  the  judgement.  Miller 
(1956),  in  his  summary  of  a  large  number  of  studies  has  shown  that  in- 


creasing  the  number  of  dimensions  operates  through  the  intervening  var- 
iable of  codability  to  improve  information  transfer  capacity  at  a  some- 
what less  than  linear  rate.   Increasing  the  number  of  alternatives,  on 
the  other  hand,  does  not  ease  codability  and  merely  adds  directly  to  the 
amount  of  information  to  be  processed.   In  this  same  line,  De  Groot 
(1964),  in  a  study  of  chess  strategies,  finds  that  grandmasters  rather 
severely  limit  the  number  of  alternative  moves  which  they  consider  in 
formulating  strategy.  Yh'tema  and  Torgerson  (1961)  in  a  discussion  of 
future  research  on  evaluative  decisions  predict  the  key  concept  to 
be  that  of  problem  simplification.   They  mention  a  number  of  possible 
strategies  for  reducing  the  number  of  facts  which  must  be  considered 
at  one  time,  and  thereby  lessening  "cognitive  strain". 

From  the  data  reported  here,  it  appears  that  engineers  operate 
to  reduce  "cognitive  strain"  through  the  heuristic  of  reducing  the 
number  of  alternative  solutions  under  consideration  at  any  time. 
Furthermore,  this  appears  to  be  an  effective  heuristic  since  the  data 
show  a  clear  relation  to  solution  quality. 

The  better-performing  engineers  in  our  sample,  when  confronted 
with  this  difficulty  managed  to  cope  with  it  through  the  heuristic  of 
comparing  two  alternatives  at  a  time,  temporarily  rejecting  one  while 
bringing  up  another  for  comparison,  thus  proceeding  through  as  many  as 
eight  alternatives  on  a  two-at-a-t ime  basis. 


SUMMARY  AND  CONCLUSIONS 

In  summary,  the  paper  has  demonstrated  the  feasibility  of  a  new 
technique  for  the  study  of  the  i  &  D  process.   This  technique  employs 
the  vehicle  of  parallel  projects  lo  provide  a  control  over  the  substance 
of  the  problem  and  to  enable  a  relative  evaluation  of  solutions.   Data 
are  gathered  by  means  of  Solution  Development  Records,  which  report  the 
probability  of  adoption  associated  with  each  solution  alternative  on  a 
weekly  basis,  and  by  before  and  after  interviews  with  the  engineers. 

The  principal  conclusions  of  the  study  are: 

1.  When  a  technical  approach  to  an  engineering  problem  be- 
comes preferred  over  any  other,  it  is  not  easily  re- 
jected; and  the  longer  it  is  in  a  dominant  position, 
the  more  difficult  it  becomes  to  reject. 

2.  Groups  producing  higher-rated  solutions  generated  fewer 
new  approaches  during  the  course  of  the  project.   There 
is  some  indication  that  these  arise  when  the  favored 
approach  encounters  difficulty,  and  are  probably  sumpto- 
matic  of  poor  early  choices. 

3.  R&D  groups  trade  alternatives  off  on  a  two-at-a-t ime 
basis.   And  the  better-performing  groups  come  closer  to 
this  ideal  than  do  their  rivals.   This  appears  to  be 
the  result  of  limitations  on  the  ability  of  humans  to 
process  information.   The  consideration  of  more  than 
two  multidimensional  alternatives  of  the  sort  studied 
may  approach  channel  capacity. 
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